\documentclass[11pt,journal,draftcls,a4paper,oneside,onecolumn]{IEEEtran}
%\usepackage{ifpdf}

\usepackage{cite}
\ifCLASSINFOpdf
  \usepackage[pdftex]{graphicx}
\else
  \usepackage[dvipdfmx]{graphicx}
\fi
\graphicspath{{./figures/}}
\usepackage[cmex10]{amsmath}
\allowdisplaybreaks[3]
\usepackage{algorithmic}
\usepackage{array}
\usepackage[caption=false]{caption}
\usepackage[font=footnotesize]{subfig}
\usepackage{fixltx2e}
\usepackage{url}

\hyphenation{op-tical net-works semi-conduc-tor}

\usepackage[T1]{fontenc}

%\newcommand{\eps}{\varepsilon}
%\newcommand{\ve}[1]{\mathbf{\bm{#1}}}
\title{Superresolution Survey}
\author{Atsunori~Kanemura%
\thanks{The author is with the Graduate School of Informatics, Kyoto
University, Kyoto 611-0011, Japan (email: atsu-kan@sys.i.kyoto-u.ac.jp).}}
\begin{document}
\maketitle
\begin{abstract}
 Superresolution methods.
\end{abstract}
\def\IEEEkeywordsname{Keywords}
\begin{keywords}
 Superresolution, image resolution enhancement.
\end{keywords}
\tableofcontents

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Introduction}

Superresolution takes a sequence of low-resolution degraded images as input,
and outputs an image with higher resolution than that of the
input~(Fig.~\ref{fig:supres}).  Such an upgrade in resolution beyond the
observation level would be useful in many fields where imaging conditions are
not ideal, such as security surveillance, satellite imagery, and astronomical
observation.  Often the resolution of acquired images is limited by the pixel
density of charge-coupled device (CCD) arrays, and other contamination due to
lens-blurring effects or shot noise may be present.  Physical or optical limits
may prevent us from removing such degradation through hardware improvement
alone; moreover, even if it is possible, it is usually costly.  An alternative
software choice is to integrate multiple observed images by utilizing signal
processing techniques.
Superresolution serves also for photo practitioners as a tool for
resolution enhancement, noise reduction, and dynamic range
increase~\cite{Gulbins:2009}.


\begin{figure}[tb]
 \centering
 \includegraphics[width=0.6\textwidth]{supres.pdf}
 \caption{Multiframe, or reconstruction-based, superresolution is the problem
 of estimating a high-resolution image from multiple low-resolution, degraded
 observations of the same scene.}
 \label{fig:supres}
\end{figure}

The literature of superresolution is vast, as can be seen from review
articles~\cite{Borman:1998,Park:2003,Choi:2004,Farsiu:2004a,Shieh:2006,%
Ouwerkerk:2006} and books~\cite{Chaudhuri:2005,Katsaggelos:2007,Bannore:2009},
each of which is whole devoted to this research topic.  The recent development
of fast computers with reasonable price is accelerating the growth of the field
of superresolution.  The survey presented in this chapter is far from
exhaustive; it aims at describing important issues and giving a fair account of
development history.

Although papers proposing superresolution algorithms make different assumptions
in modeling the superresolution problem and it is difficult to give a general
rule for classifying them, there is useful points of view from optimization
theory and statistical estimation theory.  The first axis for arranging various
superresolution algorithms is from optimization theory.  Almost all
superresolution methods can be considered to be an optimization problem of a
certain cost function subject to certain constraints.
%TODO
The second classification axis is from statistics and distinction between
maximum likelihood (ML), maximum a posteriori (MAP), and Bayesian estimation
is a useful criterion.  We can obtain an appropriate cost function that has a
natural interpretation reflecting our knowledge on images and observation
processes by utilizing the statistical estimation framework.

Of course, the function of an algorithm is important; for example, a
superresolution algorithm can be characterized by its robustness to occlusion,
its capability to preserve edges, etc.

Superresolution is closely related to (blind) image
restoration~\cite{Andrews:1977,Lagendijk:1991,Kundur:1996,Kundur:1996a}, where
we have only one observed image and do not attempt to enhance the resolution.
Image restoration does not assume multiple observations, thus there is no need
to know the relative displacements of observed images, or registration.

%A typical flow of superresolution processing is as follows.
%Multiple observations are first {\em registered}, in which the relative
%displacements of the observations are quantified.
%In the next step, the observations fused.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Early Methods}

Early attempts for superresolution dates back to the 1980s.  Superresolution
methods proposed in the 1980s and refinements of them are described in this
section.


\subsection{Frequency-domain methods}

The concept of superresolution, making use of multiple images to estimate a
high-resolution image, was first proposed by Tsai and Huang~\cite{Tsai:1984}.
They assumed that there is no blur and that the motion model is restricted only
to global translation, and utilized the aliasing relationship of Fourier
coefficients to construct superresolved images.

Kim~et~al.~\cite{Kim:1990} extended this frequency-domain method to
take into account noise, by reformulating it as a least-squares problem.  Kim
and Su~\cite{Kim:1993} have introduced regularization on the high-resolution
image and stabilized the superresolution algorithm even if a blurring effect is
present.  Bose~et~al.~\cite{Bose:1993} describes a total-least-squares
superresolution problem, or an errors-in-variable problem, that appropriately
models a situation where misregistration is present.

Formulating the image estimation problem in the Fourier domain is usually
computationally economical than spatial domain formulation.  However, the
observation model has to be spatially uniform; this constraint means situations
where the blur is space-invariant, relative motions within the observation is
translational (shift only), etc.  Therefore, it is difficult to generalize to
complicated situations where spatial uniformity does not hold.

\subsection{Spatial-domain methods}

The method by Peleg~et~al.~\cite{Peleg:1987} is one of the earliest
spatial-domain superresolution methods, in which the observation model and
registration were assumed to be known and an $L_1$ error norm was minimized
iteratively in a local and greedy manner.

Irani and Peleg~\cite{Irani:1991} have proposed the iterative back projection
(IBP) algorithm, which iteratively updates a current high-resolution image
estimate by adding an error between the observations and the high-resolution
image back-projected by the assumed observation process.  By assuming a blur in
the observation model, the IBP algorithm is capable of deblurring.  The IBP
algorithm is mostly equivalent to minimizing an $L_2$ error norm without
regularization.

Computations in the spatial domain are directly related to the real-world
geometry and are more flexible than those in the frequency domain.  One
drawback is that the requirements for the computational resources can be large.
Spatial domain formulation is widely utilized especially when complicated
considerations (such as occlusion, moving objects, or edge-preservation) are
adopted.

\subsection{Methods based on projection onto convex sets}

There are superresolution methods that have been developed based on the theory
of projection onto convex sets (POCS)~\cite{Combettes:1993,Stark:1998}.  The
method of POCS first defines convex sets that are used as the constraints on
the high-resolution image, and then an image estimate is iteratively updated by
projecting onto the constraint sets.  Stark and Oskoui~\cite{Stark:1989}
proposed the first POCS-based superresolution method, which was extended to
allow noise and blurring by Tekalp~et~al.~\cite{Tekalp:1992}.  In addition to
the blur from the aperture, Patti~et~al.~\cite{Patti:1994,Patti:1997} considered
a motion blur that can vary in time.  Eren~et~al.~\cite{Eren:1997} have proposed
a robust POCS-based superresolution algorithm that masks outlier regions in the
observations by using a validity (masking) map constructed from the
observations by thresholding the summed absolute difference in a sliding
window.  Patti and Altunbasak~\cite{Patti:2001} have introduced an
edge-adaptive constraint set on the high-resolution image, and achieved image
estimates with less ringing and more crisp edges.

Superresolution with the POCS theory is moderately popular but it seems that
only limited groups of researchers are involved in developing it.  POCS-based
superresolution methods can be seen from a more general unified
view~\cite{Elad:1997} that allows potential extensions to adopt to practical
situations.


% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Theoretical Foundation and Limits of Superresolution}

The task of superresolution may appear impossible because the increased image
resolution seem to imply that Nyquist limit~\cite{Unser:2000} is violated.
However, it has been shown both
theoretically~\cite{Papoulis:1977,Cheung:1989,Brown-Jr.:1989,Seidner:1998} and
empirically~\cite{Borman:1998,Park:2003,Choi:2004,Farsiu:2004a,Shieh:2006,
Ouwerkerk:2006,Chaudhuri:2005,Katsaggelos:2007,Bannore:2009} that
superresolution is possible if each observation contains ``novel'' information.

One of the theoretical foundation of the possibility of superresolution is
Papoulis' generalized sampling theorem~\cite{Papoulis:1977}, which guarantees
that a band-limited signal can be recovered from a set of sequences of the
original signal's linear responses, sampled {\em below} the Nyquist rate.  The
linear responses must be from a non-degenerate linear system (this condition
amounts to the ``novel'' information requirement for observations).  The
generalized sampling theorem has been extended for multi-dimensional
signals~\cite{Cheung:1989,Brown-Jr.:1989,Seidner:1998}.  Ur and
Gross~\cite{Ur:1992} utilized the generalized sampling to superresolve blurred
low-resolution images with subpixel shifts.  Generalized sampling was also
explicitly utilized in a 3D superresolution method by
Shekarforoush~et~al.~\cite{Shekarforoush:1995,Shekarforoush:1996}.  Relying on
the generalized sampling theorem, Prasad~\cite{Prasad:2007} theoretically
analyzes superresolution from observations shifted by subpixel amounts without
blurring, and describes how the determinant of the response linear system
influences the robustness of a superresolution algorithm.


Limits of superresolution algorithms are analyzed by Baker and
Kanade~\cite{Baker:2002} and Lin and Shum~\cite{Lin:2004} based on the
condition number analysis of the linear system.  Their results indicate that
there is an upper bound on the resolution enhancement factor even if we have
plenty of observed images.  This fact motivated Baker and
Kanade~\cite{Baker:2002} to invent a learning-based superresolution method.
%TODO


% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
%\section{Superresolution from a Probabilistic Viewpoint}

%Probabilistic methods~\cite{Schultz:1996} using
%Markov random fields (MRFs)~\cite{Li:2009a,Winkler:2003}.

%Cheeseman~et~al.~\cite{Cheeseman:1993,Cheeseman:1994} superresolved Mars surface
%images from Viking Orbiter, by using a MAP superresolution algorithm that
%assumed Gaussian for both the likelihood and prior.  Viking Orbiter images are
%available from NASA~\cite{NASAViking}.  In
%Smelyanskiy~et~al.~\cite{Smelyanskiy:2000}, albedos (surface reflectances) and
%heights are superresolved in a MAP formulation, which is called 3D surface
%reconstruction.  The three-dimensional superresolving reconstruction problem
%was also addressed by
%Shekarforoush~et~al.~\cite{Shekarforoush:1995,Shekarforoush:1996}.


%\subsection{Maximum Likelihood}

%Schechtman~et~al.~\cite{Schechtman:2005} have applied a superresolution
%technology to temporal resolution enhancement by using a set of observations
%with different time offsets, as well as spatial subpixel shifts.  The
%formulation of the space-time superresolution methods is based on the
%minimization of $L_2$-norm functions both for the error criterion and the
%regularization.  Increasing the temporal resolution naturally removed the
%motion blur.

%\subsection{Maximum \APosteriori}




% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Superresolution Under Standard Situations}


% Observation and regularization norm are $L_2$ unless otherwise mentioned.
The standard formulation of superresolution algorithms utilizes the framework
of regularized error minimization, which is equivalent to viewing
superresolution as a MAP estimation problem.  The error measures the fidelity
of a high-resolution image estimate to the observed data, and the
regularization penalizes unwanted solutions and stabilizes the output.

In the statistical estimation framework, the error and regularization terms
correspond to the likelihood and prior, respectively.  Defining the posterior
by specifying the prior and likelihood is a fundamental step in statistical
estimation.  Since every quantity relevant to the estimation is computed from
the prior and the likelihood, the performance of a superresolution algorithm
strongly depends on the choice of prior and likelihood models.  In the standard
formulation, the prior and the likelihood are both selected as single-layer
Gaussian distributions~\cite{Park:2003,Farsiu:2004a,Borman:1998,Tipping:2003,
Molina:2003a,Woods:2006,Elad:1997}.

As the prior, a simple Gaussian distribution can only impose smoothness
constraints over pixel values, and it is known that the Gaussian distribution
is not appropriate for natural images because it overly smoothens edges and
hence the estimated images may often be blurred.  The assumption of Gaussian
induces a $L_2$ norm or Tikhonov regularization; therefore, methods utilizing
$L_2$ regularization also have the same disadvantage.  This disadvantage is not
limited to the superresolution problem; in fact, it is shared by various image
processing problems, and considerable efforts have been devoted to develop
non-Gaussian probability distributions that can preserve
edges~\cite{Bouman:1993,Borman:1998,Park:2003,Farsiu:2004a,%
Schultz:1996,Hardie:1997,Farsiu:2004}.
Some of the studies used non-Gaussian heavy-tailed energy functions such as
Huber's robust functions~\cite{Hardie:1997,Schultz:1996} or
Laplacians~\cite{Farsiu:2004}.  Since abrupt changes in neighboring pixel
values often occur in natural images, image models that take discontinuity into
consideration will help the superresolution reproduce natural images.


As likelihood models, linear Gaussian forward models are employed in a majority
of studies~\cite{Park:2003,Farsiu:2004a,Borman:1998,Hardie:1997,Tipping:2003,
Molina:2003a,Woods:2006,Elad:1997,Schultz:1996}, whereas an L$_1$-norm-based
error term is employed in a robust superresolution algorithm~\cite{Farsiu:2004}
in order to cope with registration errors or inconsistent observations.
Contrary to the prior, the likelihood has an obvious physical counterpart;
therefore it is relatively easy to justify the likelihood model.
Gaussian models can be explained by the normality of observational noise.

As a prior, Rajan and Chaudhuri~\cite{Rajan:2002} used a compound Markov random
field (MRF) model
where line processes are introduced to account possible discontinuities in
images.  Humblot and Mohammad-Djafari~\cite{Humblot:2006} used a two-layer MRF
prior that segments a high-resolution image into regions of different
characteristics.  Both are based on the MAP estimation.
%However, they are based on pseudo-Bayesian MAP estimation and do not employ
%marginalization.  This means that their solutions are point estimates that fail
%to capture the uncertainty regarding the unobservable hidden-layer variables.
%Therefore they are not able to reap the benefits of incorporating
%uncertainty, namely, soft control of the strength of smoothing
%(Section~\ref{ssec:estep}) and
%prevention of overfitting in
%the simultaneous estimation of the registration parameters
%(Section~\ref{ssec:mstep}).
%Moreover, neither of these studies estimate the registration parameters in the
%framework of statistical estimation.

\subsection{Registration}

Image registration~\cite{Brown:1992,Stiller:1999,Zitova:2003} is an important
step for multi-frame superresolution, as accurate knowledge of the motions
within observations is critical to the performance of superresolution
algorithm.  Registration is often performed as a preprocessing step that is
independent of high-resolution image estimation.  Some superresolution methods
estimate registration parameters simultaneously with a high-resolution image
based on a unified model.

Tom and Katsaggelos~\cite{Tom:1995} used an expectation-maximization (EM)
algorithm for solving an ML estimation problem of registration parameters.
Hardie~et~al.~\cite{Hardie:1997} attempted to obtain a joint MAP estimate for
the high-resolution image and the registration parameters by minimizing the
joint posterior distribution with respect to both the high-resolution image and
the registration parameters.  They applied simultaneous registration and
superresolution to an infrared imaging system~\cite{Hardie:1998}.

The Bayesian superresolution method by
Tipping and Bishop~\cite{Tipping:2003} optimizes the marginalized likelihood of
the registration and model parameters, where the unknown high-resolution image
is marginalized out from the search space.  Gaussian distributions are assumed
to allow analytical marginalization.  This marginalization resulted in a
more accurate registration when compared with the joint MAP method of Hardie~et~al.~\cite{Hardie:1997} on the basis
of the continuous optimization.  They also estimated the width parameter of a
Gaussian point spread function (PSF) based on the same framework of
marginalized likelihood optimization.  Woods~et~al.~\cite{Woods:2006} estimated
regularization hyperparameters as well as simultaneous registration of the
observations and high-resolution image generation by maximizing the
marginalized likelihood.  The superresolution algorithm of Woods~et~al.\
performs required computations in the frequency domain and is very efficient;
but the motion model is restricted to only the translational one.

Robinson~et~al.~\cite{Robinson:2009} optimizes a joint MAP cost function using
the variable projection method~\cite{Golub:2003}, which does not involve
alternation (coordinate descent) between the high-resolution image estimation
and the registration parameter estimation.
%TODO: How about accuracy of the Robinson method?

\subsection{Uncertainty in Registration}

The high-resolution image estimate by
Pickup~et~al.~\cite{Pickup:2007b,Pickup:2007a,Pickup:2009,Pickup:2007} is the
maximum point of the marginalized posterior distribution, which is derived by
marginalizing registration parameters out from the joint posterior.  The prior
distribution on the registration parameters was assumed as a zero-mean Gaussian
distribution.  Combined with a Huber MRF image prior, marginalized estimates
relieve overly crisp edges that are observed in image estimates without
marginalization over registrations; yet they are better than blurred output by
a Gaussian prior model by Tipping and Bishop~\cite{Tipping:2003}.

% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Superresolution Under Complicated Situations}


\subsection{Dealing with Outliers through Robust Cost Functions}

Robust methods are useful, for example, when inconsistent observations are
available, when observations are occluded, or when registration is partly
wrong.  Strategies to attain robustness can be divided into two classes.  One
is to use a robust error function such as $L_1$, optimization of which
automatically discards outliers.  The other is to use a mask that hides
unwanted regions and utilizes only regions of interest for superresolution
processing.

Zomet~et~al.~\cite{Zomet:2001} has proposed to modify the calculation of the
gradient of the $L_2$ error term; they replaced the mean image appearing in the
gradient calculation by the median image.  The resultant robust superresolution
algorithm showed fairly good robustness to inconsistency in the observed image
(moving cars or walking humans in the scene), when compared with an
$L_2$-norm-minimization algorithm~\cite{Irani:1991}; and resolution enhancement
that cannot achieved by the image median or trimmed mean was observed.

The robust superresolution algorithm by Farsiu~et~al.~\cite{Farsiu:2004} uses an
$L_1$ norm as the error term, thus automatically discarding inconsistent
observations produced by e.g.\, misregistration or accidental slips.

\subsection{Outlier Rejection through Masking}

He and Kondi~\cite{He:2006a} changed the strengths of regularization for each
observed image so that observations that are not consistent with the others,
such as those specified with inaccurate registration, were suppressed in
high-resolution image estimation.

Ivanovski~et~al.~\cite{Ivanovski:2006} introduced pixel-wise selectivity where
outlier pixels are removed from high-resolution image estimation.  Outliers are
detected in an iterative manner by comparing a current estimate with the
observations.

Pham~et~al.~\cite{Pham:2008} used no regularization and a robust Gaussian-shape
cost function,
%\begin{equation}
% \rho_\text G(x)
%  = \sigma^2\biggl(1 - \exp\biggl\{-\frac{x^2}{2\sigma^2}\biggr\}\biggr)
%\end{equation}
and showed that the resulting updating rules for the high-resolution image can
be understood as weighted $L_2$ norm minimization whose weights are adaptively
calculated based on the reconstruction error.

\subsection{Superresolving Moving Objects}

There are methods that aim to superresolve occluders.
Irani and Peleg~\cite{Irani:1993} considered multiple moving objects in the
observed frames, and tried to reconstruct both the objects and background; this
is the first attempt to superresolving dynamic scenes.  Each moving object is
detected and tracked over frames, and reconstruction is performed in a way much
like the IBP~\cite{Irani:1991}, but with motion compensation.

Bascle~et~al.~\cite{Bascle:1996} superresolved a motion-blurred object in the
observed frames by tracking it and minimizing a regularized cost function.

Superresolution of the background image under moving objects was addressed by
van~Eekeren~et~al.~\cite{Eekeren:2006}.  The registration of the background
image is replaced by that of the entire image containing moving objects.
Moving objects are detected by thresholding the difference between the observed
images and the reconstructed background image degraded by the observation
model.  Superresolution of the background image is performed based on masked
observations, in a way much like Ivanovski~et~al.~\cite{Ivanovski:2006}.  The
detected moving objects are just overlaid without resolution enhancement on
the superresolved background image.

Van~Eekeren~et~al.~\cite{Eekeren:2008} considered a situation where small
objects (such as missiles) are moving in the image (``small'' was used to
mention an object whose boundary area is larger than the size of the object).
The background image is estimated by Farsiu's robust superresolution
method~\cite{Farsiu:2004}, and objects are extracted by thresholding the
difference between the downscaled background image and the observed images.
The observation process is modeled as a mixture of a background image and the
texture of foreground objects, and a cost function composed of an $L_2$ error
and bilateral total variation ($L_1$) regularization is minimized.



% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
%\section{Hyperparameter Estimation}
%
%Pickup~et~al.~\cite{Pickup:2006} estimated the threshold parameter in a Huber
%MRF and the regularization parameter by an approximate cross validation
%procedure where low-resolution pixels are randomly removed when estimating a
%high-resolution image, and the estimated high-resolution image is projected to
%the observation domain to qualify the goodness of the hyperparameters
%as the $L_1$ error of the validation pixels.  This approach is advantageous
%over a cross validation method that leaves an image out because the influence
%of misregistration can be avoided in the pixel-wise manner.


% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Learning-based Superresolution}

The superresolution methods described so far is
in particular called {\em multiframe} superresolution or {\em
reconstruction-based} superresolution~\cite{Lin:2004}.
There is another type of superresolution called {\em example-based}
superresolution~\cite{Freeman:2002}, which estimates a high-resolution
image from only one image rather than from multiple images,
based on a database developed in advance.

Recent proposals of learning-based superresolution are Baker and
Kanade~\cite{Baker:2002} and Freeman~et~al.~\cite{Freeman:2000,Freeman:2002}.
They prepare a set of training images consisting of low- and high-resolution
image pairs in advance and utilize the relationship between the corresponding
image pairs when expanding images.

Capel and Zisserman~\cite{Capel:2001a} used the basis vectors produced by
principal component analysis (PCA) on a training set of images to define
prior distributions of high-resolution images.  Two kinds of the prior have been
proposed; one is a distribution constraining images to be on the subspace
induced by PCA, and the other is a distribution that allows to vary from the
face subspace in a Gaussian-distributed way.

Inspired by Efros and Leung~\cite{Efros:1999}'s texture synthesis algorithm that
is capable of generating texture samples from given finite samples,
Pickup~et~al.~\cite{Pickup:2004} defined an image prior from a collection of
training images.  The sampled prior was chosen to be Gaussian whose mean is the
training image patch closest to the neighbor patch of a pixel in the
high-resolution image.

Resolution synthesis~\cite{Atkins:1998} (RS) uses a mixture of linear regressors
that is trained to learn the relationship between the high- and low-resolution
images in the training dataset.
Ni and Nguyen~\cite{Ni:2007} refined RS by replacing linear
interpolators with nonlinear support vector regressors.

\section{Evaluation of Image Quality}


In order to evaluate image quality, almost all papers proposing image
superresolution methods have used the mean squared error (MSE), or peak
signal-to-noise ratio (PSNR), which is a negative logarithm of the MSE
normalized by the image range.  Although the MSE is attractive since it is easy
to calculate, convenient to optimize, and justifiable from Gaussian statistical
theory, there have been criticisms of the MSE over the
decades~\cite{Girod:1993,Wang:2009b}.  Although the MSE shows good agreement
with the human visual system (HVS) in assessing the quality of images corrupted
by Gaussian noise or other apparent distortions, discrepancy between the MSE
and the recognition by the HVS has been repeatedly pointed out and alternative
measures for image quality assessment have been reported in the literature.
For example, the HVS is fairly insensitive to intensity shift or contrast
stretch, whereas the MSE is significant for such distortions.  This section
explores several attempts to developing fidelity measures that can assess image
quality in a similar way with human perception.

\subsection{Quality Assessment in General Image Studies}

Motivated by a hypothesis that the main function of HVS is to extract
structural information, Wang~et~al.~\cite{Wang:2002,Wang:2004a} have proposed
structural similarity (SSIM), a measure for image quality assessment that takes
local structure of images into account.  SSIM is robust to changes that do not
distort shape information (e.g., intensity shift and luminance stretch).

Sheikh and Bovik~\cite{Sheikh:2006a} have developed visual information fidelity
(VIF), which measures the amount of information shared between two images.  In
order to define statistical distributions that are needed to compute the mutual
information, assumptions were made on the statistics of natural images, the
model of distortions that images may experience, and the observation noise
model.

Sheikh~et~al.~\cite{Sheikh:2006} conducted a psychometric study in which human
testers were asked to evaluate the quality of presented images, and compared
the psychometric judgments with the evaluations by ten image quality assessment
algorithms.  The human evaluation and the output of the ten algorithms were
compared using the Pearson correlation coefficient, the MSE, and the Spearman
rank correlation coefficient.  VIF and SSIM were in the top group of algorithms
that were closest to the human assessment.  Statistical hypothesis testing to
ascertain the comparison was performed; however, none of the ten algorithms can
be concluded to be significant with a 5\% significance level.

Avc\i ba\k s~et~al.~\cite{Avcibas:2002} compared 26 different image quality
measures using analysis of variance (ANOVA).  Their conclusion is that although
the MSE is the best measure of the quality of images corrupted by additive
noise, other quality measures that take shape information (e.g., edges) into
account are most sensitive to structural distortion such as blurring.


\subsection{Quality Assessment in Superresolution Studies}

%Van~Ouwerkerk~\cite{Ouwerkerk:2006} uses the following three measures to
%compare superresolution algorithms: the PSNR, SSIM, and edge stability.

In this subsection, we survey studies analyzing the performance of image
superresolution methods.  A notable difference with studies on general image
quality assessment is that, in the context of superresolution, some of
situations where the MSE works poorly do not arise; for example, intensity
shift does not generally occur between original and superresolved images.  The
main source of degradation is pixelation caused by downsampling, and MSE is
able to capture it.  Nevertheless, sophisticated image quality measures are
useful in assessing the performance of superresolution methods, in that they
can measure the detectability of small structures or edge preservability,
which cannot directly be evaluable by the MSE.

Reibman~et~al.~\cite{Reibman:2006a,Reibman:2006} have collected subjective
quality evaluations of superresolved images from 22--33 human testers, and
compared them with objective measures including MSE and SSIM.  They used a
standard MAP superresolution method to recover a high-resolution image from
low-resolution, blurred, and noisy observations.  The best overall
agreement with the human evaluations was attained by SSIM, while MSE did not
fail to be consistent with the human evaluations.

Van~Eekeren~et~al.~\cite{Eekeren:2007} have compared several superresolution
algorithms by using triangle orientation discrimination (TOD)~\cite{Bijl:1998}
as the performance measure.  The high-resolution original images used by the
TOD method for evaluating superresolution algorithms are four images of
triangles pointing to different directions (Fig.~\ref{fig:fourtriangles}), and
the TOD performance is defined as the error rate of discriminating
(classifying) reconstructed images into the four classes by using the nearest
neighbor rule.  The classification criterion is the MSE between the true images
shown in Fig.~\ref{fig:fourtriangles} and the reconstructed image.  The
findings of van~Eekeren~et~al.\ indicated that the best TOD performances are
obtained when the regularization (smoothing) parameter is lower than the value
that minimizes the MSE; that is, less smooth images (but with greater ringing
effects) are advantageous for the classification task.  They claimed that the
TOD is a practically viable measure of image quality and it can be used for
predicting the performance on real-word data from the performance on simulated
data.

Pham~et~al.~\cite{Pham:2008} have proposed five objective performance measures
that are calculated only from the superresolved image, without using the true
high-resolution image; namely, 1)~SNR, 2)~edge sharpness, 3)~edge jaggedness,
and 4)~width and 5)~height (contrast) of a small blob.  Instead of using
arbitrary images (which can be feasible when the MSE is employed as the image
quality measure), Pham~et~al.\ restricted the images that are used for quality
evaluation to be ones that contains smooth regions broad enough for estimating
the noise variance, straight edges (emulating the step function) long enough
for detecting the edge spread function, and a black spot (blob) of a small size
($\text{radius} \approx 2 \text{ high-resolution pixels}$) on a white
background.  Since the true image is not available, the SNR is computed from
the estimated values of the noise and image variances.  The edge sharpness and
jaggedness are calculated for the reconstructed straight edge region; the
sharpness is defined as the width of the edge spread function, and the
jaggedness is the standard deviation of the estimated edge positions along the
straight line.  The size and height of the small blob are those of the
reconstructed spot.  Pham~et~al.\ was successful in showing the tradeoff between
noise reduction and detail preservation of superresolution algorithms.

%SSIM prefers smoother images than the MSE.


\begin{figure}
 \hfil
 \subfloat[Up]{\includegraphics[width=1in]{triangles/up.pdf}}
 \hfil
 \subfloat[Down]{\includegraphics[width=1in]{triangles/down.pdf}}
 \hfil
 \subfloat[Left]{\includegraphics[width=1in]{triangles/left.pdf}}
 \hfil
 \subfloat[Right]{\includegraphics[width=1in]{triangles/right.pdf}}
 \caption{The four triangles used for the TOD evaluation.}
\label{fig:fourtriangles}
\end{figure}
% % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % % %
\section{Other Topics}

\subsection{Mosaicing}

When we attempt superresolution, there are plenty of observations.  Then, if
the observed images are taken to cover a large scene that cannot be captured in
a single frame, it is natural to consider patch up the observations to create
an image mosaic (panorama).
Zomet and Peleg~\cite{Zomet:2000}'s proposal is to perform superresolution and
mosaicing (panorama generation) simultaneously.
They used the $L_2$ norm for both the error and regularization terms.
Capel and Zisserman~\cite{Capel:2001,Capel:2003}'s mosaicing-superresolution
algorithm uses a Huber MRF as an image prior and results in sharp
high-resolution estimates.

%\subsection{Color (or Multichannel)}
%
%\subsection{Color Demosaicing}
%
%\begin{figure}[t]\centering
% \includegraphics[width=5truecm]{survey/BayesCFA.pdf}
% \caption{Bayer color filter array.}
%\end{figure}
%
%Each pixel of a color image consists of red, green, and blur values.  Some
%digital cameras have three CCD sensors for acquiring the three channel values.
%However, some of modern color imaging devices have only one CCD sensor and use
%color filter arrays to acquire the three channel information.


%
%The Bayer array.

\subsection{Superresolution without Motion Estimation}

Protter~et~al.~\cite{Protter:2009a} have proposed a superresolution method that
does not require explicit motion estimation.  Their method is based on the
nonlocal means algorithm, originally proposed by
Buades~et~al.~\cite{Buades:2005} for image denoising.  The nonlocal means
algorithm is a kind of the moving average algorithm, but the weights are
adaptively calculated for each sliding window in a nonlocal way.
Protter~et~al.~\cite{Protter:2009a} have shown that a three-dimensional
generalization of the nonlocal means algorithm naturally accomplishes
superresolution by gathering pixel value information over the time axis.
Takeda~et~al.~\cite{Takeda:2009} have extended their steering kernel
algorithm~\cite{Takeda:2007} to three dimension to enable superresolution,
using the concept of nonlocal means.
In practical situations, it may be difficult to obtain accurate registration, and
superresolution methods without motion information would be useful.



\bibliographystyle{IEEEtranS}
\bibliography{/Users/ak/Dropbox/Papers/_akbook}

\end{document}
